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1 WIRELESS AUGMENTED REALITY COMMUNICATION SYSTEM 

2 RELATED APPLICATIONS 

3 This application is based on provisional patent application serial number 60/1 15,993 filed 

4 January 15, 1999. 

5 GOVERNMENT LICENSE RIGHTS 

6 The U.S. Government has certain rights in this invention pursuant to NAS7-1407 

7 awarded by NASA. 

8 BACKGROUND OF THE INVENTION 

9 1. Field of the Invention 

10 The invention is a wireless augmented reality system (WARS) that leverages 

1 1 communications and multimedia information processing microelectronics, along with displays, 
1? imaging sensors, biosensors, and voice recognition to provide hands-free, tetherless, real-time 

I B access and display of network resources, including video, audio and data. 

|4 2. Description of the Prior Art and Related Information 

Online instruction manuals are becoming more prevalent in the industrial and everyday 



41 environment. These electronic technical manuals (ETM) maybe interactive. Just as with printed 

If manuals, ETMs may become very difficult to use and maintain in these environments where 

;f8 elements of an environment, such as dust, chemical or general harshness may be detrimental to 

# the electronics and storage devices used to display and operate the ETM. Further, it is not always 

ifp possible for a worker who requires access to an ETM to stop work to consult ETM. 

21 These problems are multiplied in extraterrestrial environments such as a space shuttle or a 

22 space station. During intra and extra vehicular activities, it may be virtually impossible to access 

23 a traditional keyboard and computer display to access an ETM. For example, during a satellite 

24 repair mission, it would not be practical for an astronaut in a bulky extravehicular space suit to 

25 type commands on a keyboard to view a display in the extreme environment of outer space where 

26 the sun glare may make viewing impossible. 

27 Hands-free portable computers have been implemented in an attempt to solve some of 

28 these problems. For example, U.S. Patent Nos. 5,305,244 and 5,844,824 describe systems in 

29 which a head-up display and voice recognition is implemented in a portable computer for 

30 displaying ETM. However, these systems, being a single user-to-computer paradigm, do not 
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1 allow multiple-user access to multiple computers, multimedia devices or nodes on a network for 

2 accessing arbitrarily-selected data channels. Further, these previously-described systems are self 

3 contained and their data storage needs to be updated periodically to be sure that the latest data is 

4 displayed. Further, these systems do not allow two-way communication over local and wide area 

5 networks to other mutli-media users and devices, and do not provide real-time biomedical 

6 information about the physical condition of the user. 

7 There is thus a need for a wireless, wearable communications system allowing two-way 

8 voice, video and data communication between local users and to remote users and devices over 

9 network nodes, along with tetherless real-time monitoring of the local user's physical condition. 

10 SUMMARY OF THE INVENTION 

1 1 The system solves the above problems with prior art systems with an adaptive wireless 
1% remote access network comprised of small individual portable access units linked to a local 

i| cellular general purpose node. Interlinked general purpose nodes support communications across 

different habitat modules or internal-to-extravehicular communications, in the case of the space 

iliS environment; terrestrial wired networks such as the Internet can serve as the interconnection of 

jl^ remotely scattered access nodes in an industrial, commercial or home environment application. 
17 The system may provide shuttle and international space station astronauts with tetherless, 

115 on-demand access to data channels from multimedia devices such as cameras or audio sensors 
l W associated with other persons or in a stand-alone configuration, and multimedia or data display 
W from a networked computer terminal and to the equipment control capabilities which may be 

21 available through that computer. Transparent to such access, the system can maintain a data 

22 channel for monitoring an astronaut's health or environment via in-situ sensors. Though this 

23 system may be used for the shuttle and the international space station, the system has uses in 

24 many possible applications related to medical, industrial, and commercial areas. 

25 The invention is a personal communications system designed especially for the space 

26 shuttle or station environment to provide cellular communications access throughout the vessel 

27 with video, audio, data and computer connect services. A small, wearable portable access unit 

28 (PAU) communicates over high-rate link to a centrally-located network access unit, called a 

29 general purpose node herein. The system backbone provides 2-way video, 2-way audio, and a 

30 multi-purpose data channel between the PAU and general purpose node. One embodiment of the 
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1 PAU used for personal communication has an attached headset with video display, audio feed 

2 and camera, which together may be used for audio or video teleconferencing. When used as a 

3 virtual terminal to a computer in the network, the user is able to view and manipulate imagery, 

4 text or video, using voice commands to control the terminal operations. 

5 Using the system, an astronaut may efficiently operate and monitor computer-controllable 

6 activities inside or outside the vehicle or station. Hands-free access to computer-based 

7 instruction texts, diagrams and checklists replaces juggling manuals and clipboards, and 

8 tetherless computer system access allows free motion throughout a cabin while monitoring and 

9 operating equipment. Along with voice commands, an integrated "touchpad" on the PAU may 

10 be used for remote computer control through a sensor data channel; this return data channel may 

1 1 also be used for other control data as from a three-D mouse or data glove input device, allowing 
If the real-time video display to be used for remote, wireless monitor and control of robotic 

li cameras or manipulators. 

14 Concurrent with information provided to the astronaut, the system also allows external 

t§ observation of the astronaut's situation; personal biological or other sensors can send back 

1% continuous telemetry through personal access unit and general purpose node. A miniature 

17 camera integrated into the headset provides real-time video of the wearer's field of view to 

pL8 remote observers. In this way, for example, a principal investigator located on Earth may consult 

j L9 with a payload specialist on the operation or troubleshooting of their equipment. 

Wd The system provides wireless high-rate data exchange. The radio link is adapted to 

21 operate within a high-interference, high-multipath environment of a space shuttle or space station 

22 module. Radio frequency (RF) links do not require visual line-of-sight to operate, but the metal 

23 walls and lack of RF absorbers, combined with moving human bodies, creates an enormous 

24 potential for destructive self-interference of the radio signals. The integrated radio and 

25 multimedia data processing technology provides for efficient and high-quality video and audio 

26 data compression for noisy indoor communications channels. The system supports multiple-user 

27 access for video, audio, and sensor data services in the wireless coverage area. 

28 Potential applications of the system are in any environment where heads-up, hands-free 

29 information retrieval or multimedia communications access improves efficiency including 

30 tetherless operations/monitor consoles, remote consultations in medical or maintenance 
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1 procedures, and hazardous/confined space activities. There are also in-the-home 

2 entertainment/communications applications. 

3 Similar to the space extravehicular activities applications, bio-isolation suits have similar 

4 operation constraints to space suits. They are worn commonly where there are chemical or 

5 biological contaminates, and any extraneous materials brought into a chamber, such as clipboards 

6 or documents, also present a contamination risk. A unit suitably modified for use in a space suit 

7 could be used in this situation. This allows the user to use a computer (log data, use a check list, 

8 etc.), to communicate with colleagues, including providing first-hand video of work in progress, 

9 and to maintain constant monitoring of the health of the user. 

10 An extension of the medical applications areas would be in remote telemedicine. Many 

1 1 medical diagnostic and treatment tools are being made portable and rugged enough to be taken to 
1% remote sites. Some examples are an ultrasound unit that is the size of a backpack, an entire 

l| intensive care unit of equipment built into a stretcher, and a trauma pod built into a cruise 

44 missile. For many of these devices, CRT or LCD panels comprise a significant amount of the 

JJ bulk and weight of the devices. The system of the present invention may provide a replacement 

ijj for the CRT or LCD panel as well as an interface to the control system of the device, while 

47 providing communications access through an interface to the remote site's existing 

ills communications equipment. 

% Industrial applications include use by inspection or maintenance crews in remote or 

iS) dangerous environments such as oil refineries, drilling rigs, power plants, etc., where the 

21 personnel can move around with their hands and peripheral vision free to attend to their own 

22 safety and tasks. They would be in constant contact with the information they needed and any 

23 technical assist could be given by individuals looking at the return video images from the user. 

24 An example of a commercial application is for mission control and other operations 

25 personnel who presently must sit at a display console for hours at a time. These individuals 

26 could make use of the system of the present invention to increase their mobility and efficiency. 

27 BRIEF DESCRIPTION OF THE DRAWINGS 

28 Fig. 1 is a diagrammatic illustration of the components of the system of the present invention; 

29 Fig. 2 is block diagram illustrating communications components used by the personal access unit 

30 and general purpose node of the system of Fig. 1 ; and 
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1 Fig. 3 is a flowchart illustrating a method performed using the system of Fig. 1 . 

2 DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

3 With reference to Fig. 1, a diagram illustrating components of the system of the present 

4 invention is shown. The system may comprise small pager-like devices called portable access 

5 units 100. The portable access units 100 are accessorizable for different "multimedia" interfaces 

6 for display, camera, audio and sensor operation. Another embodiment of the portable access unit 

7 1 00a comprises a wearable headset and microphone assembly 102a. 

8 The portable access units 100- 100a interface directly through wireless link with a network 

9 through a general purpose node 150. The general purpose node 150 allows wireless-to-wire 

10 communication with a local network 170. The local area network 170 may be electrically 

11 connected to a wide area network or Internet 172 in order to connect to remote local area 
J;2 networks 174. Alternatively, the general purpose node 150 may be directly connected to the 
i§ wide area network 1 72. The general purpose node 150 may thus act as a router for routing video, 
It display, audio and control data packets between the portable access units 100 and other, or 
|| remote, portable access units 100 or remote media devices 125, 180, etc connected to the 
j| networks 170-174. The connection with a network 170-174 may occur directly in electrical 
17 connection with one of the networks 170-174, or in wireless communication through a remote 
p5 general purpose node 150a that is electrically connected to the network. The portable access 
j i3) units 100 may provide communication to and from remote media devices comprising computers 

IKS* 

£3) 180-182 running specialized client software or certain commercial multimedia Internet software 

21 products such as video conferencing products that adhere to the industry standard H.323 for 

22 multimedia data transfer. 

23 Each portable access unit 100- 100a may dynamically associate with the closest general 

24 purpose node 150-1 50a when it is logged on to the network 170-1 74 or is connected thereto. 

25 Each general purpose node 150- 150a records the associations and registers each portable access 

26 unit 100-100a on a list of connections associated with the particular general purpose node 150- 

27 1 50a. The list of connections is stored in a random access memory device included in the general 

28 purpose node 150-1 50a. When a portable access unit 100 is logged off or disconnected from the 

29 network 170-174, it is disassociated from the particular general purpose node 150-150a that it 

30 was associated with, and thus, is removed from the list of connections. 
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1 As shown on an example selection list screen 190 that may be presented on a display 102 

2 or headset 102a on any of the portable access units 100-100a, the user can set up a video, audio, 

3 or data link with any other portable access unit 100-100a or remote media device 125, 180, etc, 

4 that is logged onto a network 170-174. The headset 102a may comprise a heads- up display (120 

5 in Fig. 2) inside a headset embodying a transparent color LCD device. Using control keys or 

6 voice commands, a user of the portable access unit 1 00- 1 00a may select a local or remote 

7 portable access unit 100-100a on a selection list 190 of other portable access units 100-100a or 

8 media devices 125, 180. The selection list 190 comprises a combination of the lists of 

9 connections stored on all of the general purpose nodes 150-1 50a. Users may further access a 

1 0 nameserver located on the access node 1 50 for locating remote unfamiliar portable access units 

11 100-lOOa or remote media devices. 

12 By selecting entries from the selection list 190, users may communicate with portable 

H access units 100-lOOa or various media devices such as cameras 125, internet phones 104, one or 

m more computers 1 80-1 82 located throughout the networks 170-174. A user may further select 
Jl from the list 190 user names representing users of other portable access units 100 that are logged 
il^ in and associated with remote general purpose nodes 1 50a connected to the networks 1 70- 1 74. 
1 7 With reference to Fig. 2, the components of the access node 1 5 0 and the wearable headset 

lli embodiment of the portable access unit 1 00a is shown. Elements for both the general purpose 
'19 access node and portable access unit 100a include a communications device 202. Data 
ib processing functions are implemented in the form of an audio/video coder/decoder (codec) pair 

21 200, one codec 200 comprising part of the portable access unit 100a and the other codec 200 

22 being part of another portable access node 1 00a or remote media device for which it is desired to 

23 exchange signals. At a portable access node, the codec 200 controls a digital data stream which 

24 is fed to the communications device 202, which is implemented as an RF modem transceiver pair 

25 with an equivalent communications device 202 located in the general purpose access node. The 

26 codecs 200 serve as the interfaces to the external elements (including possibly the user display 

27 1 02a and the sensor 1 04) on both sides of the communication continuum comprising the 

2 8 communications device 202 of the general purpose node 1 50, an internal network interface 

29 protocol circuit 152, the external networks 170-174 and the electrical connection or general 

30 purpose access node connection to the desired remote portable access node or media device. . 
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1 The internal network interface protocol circuit 152 may comprise an Ethernet chip, memory and 

2 a network protocol chip. With this architecture, the system addresses the issues of multiple- 

3 access and data channel quality, through the implementation of the communications device 202. 

4 Multiple implementations of the communication device 202 in the general purpose node 150 

5 allows for multiple simultaneous communication links with a plurality of portable access units 

6 100-100a for the general purpose node 150. 

7 With the base functionality of the communications device 202 and codec subsystem 200, 

8 the architecture provides flexibility in utilization of different external components such as 

9 different headset 102a configurations, sensor 104 packages, and network interface 152 

10 capabilities. 

1 1 The communication device 202 is designed to operate in a high multipath space station or 

12 terrestrial indoor environment while being able to support multiple users at high, multimedia- 
type bandwidths. Thus the communications device's 202 physical (PHY) and media access 

14 (MAC) layers in combination support multiple access, dynamic network association, channel 

15 error rates of broadcast video quality (lxlOe-6), and data rates up to broadcast-quality video 
jiB bandwidths (on the order of 768 kbps per user (one-way)). Modulation to achieve this 

17 performance will be differential phase-shift keying, of binary or higher order (quadrature or high- 
lit order quadrature amplitude modulation); the order chosen reflects the necessary user data volume 
© to be supported within fixed, FCC-specified bandwidth allocations. Orthogonal frequency 



£0 division multiplexing, code division multiple access, and frequency hopping/time division 



22 equalization, antenna diversity and retransmission techniques may also be used for improving the 

23 reliability of the communications link. Through a combination of these technologies, two-way 

24 multimedia channel throughputs can be achieved for each of multiple simultaneous users. A 

25 variety of RF frequencies may be used, but the determining factor in frequency band selection is 

26 the availability in the band of a relatively large amount of spectrum in the space station or FCC 

27 terrestrial allocations, allowing the transmission of compressed video. Ranges in the 2.5 to 5.7 

28 band range are preferable due to the FCC bandwidth available, the compactness of RF elements 

29 required at these frequencies, and the potentially low amount of interference that will be 

30 sustained. The RF front end of both the portable access unit 100- 100a and general purpose node 
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1 1 50- 1 50a may be interchangeable with different frequency front ends for system use in different 

2 frequency bands. 

3 Low-rate, single user implementations of the communications system may be effected 

4 through adapted commercial wireless-LAN type products following the FCC 802. 1 1 standard 

5 such as a frequency-hopping 2.4 GHz wireless LAN transceiver by Waveaccess , Inc of 

6 Wellesley, MA, or direct-sequence spread-spectrum 2.4 GHz wireless LAN chipset by Harris 

7 Prism of Melbourne, FL. These radio implementations, as with commercial implementations of 

8 the industry-proposed Bluetooth and HomeRF standards, will be limited in user access and 

9 overall throughput, however, and therefore unsuitable to real-time video teleconferencing for 

10 multiple users. The preferred embodiment for full capability is to implement the 

1 1 communications devices' physical and media access control layers in custom ASIC circuits 

1-2 allowing for support of all system capabilities within microelectronics architecture for small size 

ii and low power draw, providing pager-type form factor of wearable personal access units 100- 

m 100a. 

i|| The communications device 202 comprises a buffer memory and a radio frequency front 

:i| end. Data modulation/demodulation circuits and error detection/correction protocol circuits are 

I.. : ; 

17 further included. Various combinations of these circuits may be obtained from Proxim of 

jLB Sunnyvale, California, Harris of Melbourne, Florida and Stanford Telecom of Stanford, 

IP California. Alternatively, all of the various circuitry may be implemented with an application 

03) specific integrated circuit (ASIC), or a combination of an ASIC and discrete elements for size 

2 1 and weight efficiency. 

22 Three classes of headsets 102a may be used: hi-resolution military systems which are 

23 CRT based and may be provided by Honeywell of Morristown, New Jersey, or Hughes Network 

24 Systems of San Diego, California; medium resolution industrial systems which are CRT or LED 

25 based scanners and may be provided by Intervision of Santa Clara, California; or low to medium 

26 resolution entertainment systems which are color viewfinder LCD based systems that may be 

27 supplied by Virtual Vision of Redmond, WA (the V-CAP and E-GLASS), Sony Europe of 

28 Hampshire, United Kingdom (GLASSTRON VISOR) or Olympus of San Jose, California. 

29 Typical headset display 120 specifications for the portable access unit 100a include the 

30 following: 



8 




# 



PATENT 
Docket No. 9373-1 F888US1 

1 - RESOLUTION: Comparable at least to VGA (640x480) or better to 1 280 x 1 024 

2 w/ofF-the-shelfdisplay&I/Oconfigviration 

3 - DISPLAY: >10 FL/day,Display Bright.Ratio:>2,Brightness range:2 OOMmax 

4 - FOV: 40-60 deg ,Gray scale: > 1 2 

5 - EyeRelief: 20-26 mm TSP, 14/10mm(on/off-axis) exit pupil 

6 - Unif: 2: 1 across 2/3 FOV, GLARE:< 2.5% image content,PixelContrast:25 

7 - FOCUS: Hands off, Obs:, % look- around ,Diopter range: ±2, 

8 - Mag: 1± p5%,Cont: >95%, motion sensor 10°cone, Inter. Eye. adj: 52-72mm 

9 - Image Enhan & IFF: Weaponsight, motion sensor and computer interface 

10 The audio/video codec 200 in a portable access unit 100- 100a or other client device is 

1 1 based around a single chip, standards- based codec that accepts analog or digital audio and video 

12 (i.e. NTSC or VGA) compresses this input, and multiplexes the compressed data with an external 
1$ data stream. The preferred industry standards are: ITU H.263 based video, ITU G.722 based 

ia audio, and ITU H.221 based multiplexing. The audio video codec 200 in the portable access unit 

'fs 1 00- 1 00a can establish a link with a similar audio/video codec 200 associated with another 

ft portable access unit 100-100a or a remote media device 104, 125, 180 or 182. The signals from 

17 the codec 200 in the portable access unit 100a outputs the received and decompressed remote 

signals from the device for which the link was established. The interface between the codec 200 

j tb and communication device 202 as well as between the communication devices 202 of the general 

S5) purpose node 1 50-1 50a and portable access unit 100-1 00a operate two-way with a high 

21 bandwidth suitable for transmitting video. Of this bandwith, the audio portion utilizes up to 64 

22 kbps and the data from the sensor 104 utilizes the required amount for the type of sensor 104, 

23 with the remainder allocated to compressed video. The quality of the video at these data rates in 

24 excess of 128 kbps is at least equivalent to video teleconferencing quality video. 

25 The audio/video codec 200 portion of the portable access unit 100- 100a may further 

26 comprise video input and output ports, audio input and output ports, data input and output ports, 

27 and a the above-mentioned multimedia processor chip for packaging signals for data 

28 compression and decompression for transmission. Exemplary multimedia processors include the 

29 VCPEX chip by 8x8, Inc. of Santa Clara, California or digital signal processing chips by Texas 

30 Instruments and others. The audio/video codec 200 further comprises a field processor gate 
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1 array, electrically programmable read-only memory and random access memory for processing 

2 and packaging signals for compression and decompression 

3 The sensor 1 04 may comprise a commercially available pulse oximeter sensor or other 

4 type of bio-sensor. A pulse-oximeter sensor allows the measurement of pulse rate and oxygen 

5 saturation of the blood. Data from the sensor 104 is transmitted to the general purpose node 150- 

6 1 50a, and transmitted to any remote media device connected to any of the networks 170-172. 

7 The sensor 104 may comprise an "on body" wireless human performance and fatigue monitoring 

8 system that communicates with a belt-mounted transceiver/control module. The remote media 

9 device may comprise a processor 180-182 for display or logging of the real-time sensor signals. 

10 The headset 102a comprises a heads-up display 120 comprising a transparent color LCD 

1 1 device for video signals received and processed by the codec 200. The headset 102a may further 
\% comprise, or have attached thereto, an integrated microphone 122 for receiving voice commands 
it from the user of the portable access unit 100a or for communicating voice signals to a remote 

1% portable access unit 100 or remote media device. The headset may further comprise a speaker 

Lii 

15 124 or earpiece unit for presenting audio signals to the user. The portable access unit 100a may 

ijij) further comprise a digital camera 106 that may either be attached on the user's person, or to the 

17 headset 102a for providing video signals to other portable access units 100- 100a or media 

M devices. 

19 With reference to Fig. 3, a flow diagram illustrating the method performed by the system 

If) of Fig. 1 is shown, A user puts on the headset 102a, portable access unit 100a, step 400. The 

21 user may log into the local general purpose node 150 wherein the portable access unit associates 

22 with the general purpose node 150 such that the user is added to a connection list stored in a 

23 random access memory device residing in the general purpose node 1 50, step 401 . Data is 

24 provided from the general purpose node 150 to the portable access unit through the 

25 communication devices 202, step 402. The user is presented with a selection list 190 of portable 

26 access units 100- 100a and media devices logged onto the system on the display 120, step 404. 

27 The user selects one of the entries from the selection list, step 406. The selection is transmitted 

28 to the general purpose node 150, step 408. The general purpose node 150 sets up a connection 

29 over the networks 170-174 for channeling data between the portable access unit 100a and the 

30 selected network device, step 410. The selected network device may comprise the processor 180 
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1 or other network client 182 for running a software application, a camera 125 for providing 

2 remote viewing operations to the user on the display 120, the Internet phone 104 for providing 

3 voice communications with the a remote user, or another portable access unit 100- 100a over a 

4 remote general purpose node 1 50a. By providing control commands to the microphone 122 or 

5 other input system, such as a keyboard or handheld mouse, the user may conduct operations by 

6 transmitting commands between the portable access unit 100a and the general purpose node 150 

7 which routs the control commands to the device that the user selected, step 412. 

8 It will thus be seen that changes may be made in carrying out the above system and method 

9 and in the construction set forth without departing from the spirit and scope of the invention, it is 

1 0 intended that any and all matter contained in the above description and shown in the accompanying 

11 drawings shall be interpreted as illustrative and not in a limiting sense. 



